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This study was designed to determine the degree of use, level of client satisfaction of professional 
development and educational sendees, and to identify suggestions for improving sendees. Results from a 
mixed methodology approach indicated moderate to high levels of satisfaction in two program areas and 
moderate to high levels of dissatisfaction in a third program area. Responses to open ended questions 
revealed specific rationale concerning dissatisfaction in technology support and four professional 
development areas. 
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Twenty-five percent of Americans are enrolled in educational institutions, with many completing bachelors, masters 
or doctoral degrees to prepare themselves for teaching positions (United States Department of Labor, 2005). 
Continuing education of these professional educators is increasingly critical, especially when knowledge doubles 
every three years or less (Alstete, 2006). Organizations that offer continued professional development and support 
for educators are labeled as “educational services” by the Department of Labor. America’s second largest industry, 
educational services, accounts for about 13.0 million jobs. Educational service agency clients often seek assistance 
for meeting accreditation standards and equalizing educational opportunities, the effective use of educational 
resources, technical instruction, and other services. These organizations may also coordinate services between State 
Departments of Education and local school districts. Clients are often teachers, teacher’s aids, school 
superintendents, principals or assistant principals, and administrative and other supportive staff (State of Arkansas, 
Bureau of Legislative Research, 2006). 

Determining how to ensure high-quality services at each of Arkansas’ 15 independent Educational Service 
Cooperatives (ESC) is “no easy task and is like herding cats” (Blomeley, 2006, September 15). As recently as 2003, 
the former Governor, Mike Huckabee, proposed that the state should assume full responsibility of these service 
centers. Though several of Arkansas’ ESCs directors claim their establishments provide valuable services to school 
districts in their respective regions, the state’s leaders remain concerned about improving the quality of education 
and the need to build and train an educated workforce. Compliance with the No Child Left Behind Act of 2001 
continues to be a major focus for education providers throughout the United States. The Act established Federal 
guidelines and provided financial assistance to the states to ensure that all students in public elementary through 
secondary schools receive an equitable and high-quality education. To avoid the loss of Federal funding, states must 
standardize testing of all students in core subject areas. While the No Child Left Behind Act is federally mandated, 
states and local stakeholders maintain the freedom to allocate funds to the most needed areas (No Child Left Behind, 
Act of 2001). The rationale for creating a network of ESCs in Arkansas was to facilitate the professional 
development of the school districts’ staff and ultimately increase the quality of teaching and student performance. 
The ESC’s main duties are “providing professional development for teachers in the districts and to offering 
technology training” (Blomeley, 2006, September 15). Approximately $78,130,598 in total revenue was reported by 
the 15 education service cooperatives in Arkansas for the year ending June 30, 2006. Approximately 50 percent of 
that revenue was attributed to state assistance, 26 percent to federal assistance, and 24 percent considered local 
revenues (State of Arkansas, Bureau of Legislative Research, 2006). At present, there has been no comprehensive 
evaluation of the cooperative’s program of services published within the literature. The purpose of this research is to 
review an aspect of the evaluation process by sharing the results of a recent client survey from more than 6,600 
stakeholders. Further, a goal of this paper is to describe the limitations of this one -dimensional model as a decision- 
making tool compared to how this approach differs from the ideal form of program development and evaluation 
offered in the logic model (Fitzpatrick, Sanders, & Worthen, 2004). 
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Problem Statement and Study Objectives 

Educational Service Cooperatives consume valuable financial, time, and human resources. It is therefore important 
to periodically ascertain their value through evaluation. Empirical research is needed to show the impact of ESCs 
and help decision makers determine whether “programs should be continued, improved, expanded, or curtailed; to 
assess the utility of new programs and initiatives; to increase the effectiveness of program management and 
administration; and to satisfy the accountability requirements of program sponsors” (Rossi, Lipsey, & Freeman, 
2004, p. 2). This research represents a follow-up to an earlier study authorized in 2002 (College of Education and 
Health Professions, University of Arkansas, 2003). The study sought to achieve the following objectives: (1) 
Determine the degree of user utilization of products and services, (2) Determine user satisfaction of products and 
services, (3) Identify client suggestions for improving or expanding educational services that the ESC offer. 

Theoretical Framework 

The logic model of evaluation as advanced by Fitzpatrick, Sanders, and Worthen (2004), provides the theoretical 
framework for this study. The logic model of evaluation is a framework that represents the linkages between the 
project inputs, activities, outputs and the short-term and long-term outcome (Preskill & Russ-Eft, 2006). Based on 
the implementation structure of the ESCs in Arkansas, the logic model of evaluation provides a map of the ESC’s 
flow of inputs, activities, outputs, and the return on investment for the 78 million dollars that the federal, state and 
local revenue sources have invested in this educational support system. The logic model of evaluation seeks to 
specify the project’s processes that ultimately impact its outcomes. The primary purpose for the entire ESC system 
is to provide support to teachers through professional development activities, technology training and assistance, and 
to provide necessary resources to increase the quality of teaching and student performance in the entire state. 

The ESC system’s processes for achieving its long-term goal/outcome begins with the inputs into the system, 
which include the funds from the federal agencies, state and local sources, the time spent on training and other 
professional development activities, human and other resources invested in the project. The next component of the 
logic model specifies the activities, which include literacy programs; K-12 Maths program/training, and other 
professional development programs offered by the ESCs. To determine goal achievement of activities, both 
formative and summative evaluations should be conducted at each stage of the evaluation process to determine the 
success of each activity. The outputs in the logic model focuses on the units of service that result from the activities. 
The final set of components in the logic model detail the outcomes of the project, which are viewed in terms of 
short-term and long-term outcomes. Outcomes in this study represent the changes in the participants’ perception 
regarding the effectiveness of the ESC project. The results of this study are based solely on the summative 
evaluation of the respondents’ satisfaction with the ESC’s services. While the results of this survey may be useful 
for administrative purposes, to fully examine the ESC’s operation and the degree to which they are currently 
achieving their primary goal of facilitating the professional development of the school districts’ staff to increase the 
quality of teaching and student performance, a comprehensive and systematic evaluation based on the logic model is 
recommended. 

Literature Review 

Evaluation is an immensely important component of accountability, yet it lacks a universal definition among the 
community of professional evaluators. Fitzpatrick et al. (2004) noted, “The term evaluation has been used rather 
broadly without definition beyond what was implicit in context” (p. 4). Fitzpatrick et al. (2004) defines “ evaluation 
as the identification, clarification, and application of defensible criteria to determine an evaluation object’s value 
(worth or merit) in relation to those criteria (p. 5). Rossi, et al. (2004) explained that the evaluation of social 
programs should answer questions related to the nature and scope of the problem, its location, the stakeholders and 
numbers affected, and specifically how the problem affects them. Caffarella (2002) stated “program evaluation is 
most often defined as a process used to determine whether the design and delivery of a program was effective and 
whether the proposed outcomes were met” (p. 225). Further, multiple approaches may be utilized in combination 
during the evaluation process (Bledsoe & Graham, 2005; Chavis, 2004; Scriven, 1997). Patton (1997) placed an 
emphasis on the systematic collection of information, while Preskill and Torres (1999) focused on “evaluative 
activities specifically conducted within organizations for the purpose of organizational learning and change” (As 
cited in Preskill & Russ Eft, 2006, p. 17). 

While the client satisfaction survey data collected in this study is helpful for administrative services, and is 
considered an aspect of evaluation (Kirkpatrick, 1998), these researchers do not suggest that this research is as a 
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comprehensive evaluation as defined by professional evaluators (Patton, 1997; Rossi et al., 2004). Further, the 
findings of the client satisfaction survey revealed respondents' perceptions on various service -related items, but 
lacks the outcome component of a bonafide evaluation (Hamm, 1988; Holton, 1996). To obtain an accurate 
evaluation, “outcomes need to be examined in relation to program inputs (or measures of services) to determine the 
effects of the services” (State of Arkansas, Bureau of Legislative Research, 2006). 

The Educational Sendee Agency 

As noted by the Association of Education Service Agencies (AESA), ESAs exist in 42 states in America and 
were created by “enactment of special state legislation or administrative rule to provide programs and services to a 
collection of schools and local school districts or to serve state interests in other ways” (Stephens & Keane, p. 1, 
2005). Though invisible to many, including some state legislators, ESAs are directly or indirectly responsible for the 
expenditure of billions of dollars and may represent the largest number of service centers in America. AESA 
estimates of 1.5 trillion dollars (primarily for K-12 issues) may flow through these centers, and this figure may not 
include other services offered by ESAs (Stephens & Keane, 2005). However, due to difficulties in gathering data it 
is uncertain exactly how much the expenditure levels are in the aggregate form. It should also be noted that some ad 
hoc programs that provide a single program or service are not recognized by the AESA, the national professional 
association representing service agencies throughout America (AESA, 2000). 

Arkansas’ Education Sendee Cooperatives 

In the early 1980s Arkansas State legislators approved the funding for eight pilot educational cooperatives 
governed by local boards consisting of representatives of the school districts served. The cooperatives were 
independent and not considered an arm of the State Department of Education. Subsequently, a study committee 
gathered survey data from superintendents, teachers, and parents. Their recommendations would later lead to the 
establishment of seven additional educational service cooperatives covering all regions of the state (Arkansas State 
Board of Education - Statewide Study Committee for Pilot Cooperatives, 1984). Currently, 15 education service 
cooperatives exist throughout the state. “The cooperatives serve to provide support to the school districts in their 
region, as well as provide professional development opportunities and act as a consortium for purchasing certain 
services and supplies. The cooperatives also provide technical computer support services to the schools in their area” 
(Arkansas Department of Education, ( |[ 1, 2007). Specific examples of some of the support services offered by the 
cooperatives are listed as follows; literacy programs, math programs/training, early childhood programs, gifted and 
talented services, health and wellness services, special education services, instructional distance learning for 
students, teacher center services, and administrative support services for superintendents and principals. 

Research Methodology 

Target Population 

The potential respondents to the survey consisted of all clients from the 15 education service cooperatives 
located throughout the state of Arkansas. Clients consisted of superintendents, office staff, principals or assistant 
principals, teachers, and classified staff. The web survey aimed at collecting data from 10,000 clients. When the data 
collection period of two months was over, 6,698 clients had responded to the online survey (66.98 % of the 10,000) 
with 6,629 respondents completing usable surveys. 

Instrumentation 

A research questionnaire entitled “Arkansas Education Service Cooperatives Client Survey” was designed 
based on a review of the literature and previous studies conducted in the area of education services. The survey 
questionnaire included information related to demographics, utilization of cooperatives services, and reasons for not 
utilizing the services. The respondents were also required to rate their level of satisfaction with the services provided 
on a Likert -type of scale with the items - very satisfied, satisfied, not satisfied, and n/a. In addition, the instrument 
had open ended questions that were used to gather qualitative data to support the quantitative data obtained. The 
integration of both qualitative and quantitative data was used to enhance the interpretation of the survey results. The 
integration of multiple data sets is common to mixed method research design (Creswell, 2003). The content validity 
of the instrument was established by research experts in the field of research methods for review. The directors of 
the cooperatives also reviewed the instrument for face validity and content validity. Prior to data collection, a pilot 
study was conducted. Five people in the similar population were requested to complete the instrument. Care was 
taken to ensure that the five did not participate in the final study. Comments obtained from the directors and the 
input from the five respondents were incorporated and helped in preparing the final instrument. The final instrument 
for data collection was developed by the research faculty involved with this project in collaboration with the 
Education Service Cooperative Directors. 

Data Collection and Analysis 
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Prior to data collection, permission to collect data from the clients was obtained from the Institutional Review Board 
(IRB). Each administrator of the 15 education service cooperatives informed their local clients about the online 
survey and encouraged them to voluntarily participate. Over a period of six weeks, ESC clients accessed the online 
link and completed the surveys that were archived in an electronic database. Each survey was screened for errors 
and 39 surveys were omitted due to incomplete responses or excessive voids. The quantitative data were analyzed 
with SPSS software. The qualitative data were analyzed using a modified quasi-statistical content analysis. This 
process included reviewing each respondent's answer to the survey’s two open-ended questions. These two 
questions were inserted into the survey so respondents could provide specific information and examples regarding 
the ESC’s quality of service. The first open-ended question asked respondents for specific changes or improvements 
and to offer suggestions regarding how those services could be improved. The second question asked respondents to 
list four areas of professional development that they would like the ESC to deliver. Unlike the survey’s quantitative 
counterpart, textual material from these questions was analyzed to provide evidence that explained and supported 
the quantitative findings. The first step of the content analysis included reviewing each individual’s written 
responses to the two questions. The researchers reviewed the text of each participant’s response to identify 
commonalities among all responses. Individual words from the text were grouped into various categories and 
further grouped into the predominating themes (Creswell, 2003). 

Findings 

Demographics and Utilization of ESC Sendees and Program 

There were 6,698 respondents (clients) who accessed the online survey and 6,629 that completed usable 
surveys. The majority (82.6%, n = 5535) were teachers, 6.7 percent (n = 449) were principals or assistant principals, 
1.4 percent (n = 93) were superintendents, 4.55 percent (n = 305) were classified workers, and 3.69 percent (n = 
247) were affiliated with the main office. The data revealed that 43.7 percent (n = 2,932) were employed more than 
10 years, 23.92 percent (n = 1,602) were employed for a period from 5 to 10 years, 17.26 percent (n = 1,156) were 
employed from 2 to 4 years, and 14.5 percent (n = 971) were employed less than 2 years. Further, 61% (n = 4, 089) 
were employed as elementary, junior high, and middle school teachers. Approximately 25 percent (n = 1,650) were 
employed as high school teachers. Regarding utilization of ESCs services, forty percent (n = 2,688) of the 
respondents frequently utilized the services, 51.1 percent (n = 3, 425) utilized the ESCs services 3 to 4 times per 
year, while less than 8 percent (n = 508) never used the ESCs services. Approximately one percent (n = 77) elected 
to not respond. 

Satisfaction with Professional Development Sendees and Programs 

Table 1 shows the level of satisfaction regarding services for professional development provided by the ESCs in 
three categories: (1) Seven Literacy Programs; (2) Five K-12 Maths Program/Training; and (3) Seventeen Other 
Professional Development Programs. The majority of clients were very satisfied with the level of service provided in 
professional development programs in Reading Recovery (70.0%, n = 4,688), Early Literacy Learning (ELF) 
(66.1%, n = 4,429), Next Step Strategies (65.2%, n = 4, 367), Coaches Training (67.9%, n = 4,549), and Reading 
First (68.4%, n = 4,482). A similar level of satisfaction is reflected in Table 1 regarding the perception of 
professional development services provided for the K-12 Maths/Program Training items. For example, clients were 
very satisfied with MathLinks (68.5%, n = 4,582), Geographic Information Systems (76.0%, n = 5,092), Math 
Coach/Training (71.6%, n = 4,795), and Algebra, Geometry (71.0%, n = 4,758) programs. Clients’ perceptions in 
nine of the Other Professional Development Programs ranged from 50.6 percent to 65.7 percent indicating a 
moderate to high level of satisfaction for professional development services associated with the Early Childhood 
Programs, Career and Technical Education Services, Gifted and Talented Services, Health and Wellness Services, 
Special Education Services, ADE Professional Development Through Distance Learning, Instructional Distance 
Learning for Students, Services for Superintendents, and Assistance with Data and Support of Arkansas 
Comprehensive School Improvement Plan (ACSIP). However, clients’ perceptions in nine of the professional 
development programs ranged from 44 percent to 55 percent indicating a moderate to high level of dissatisfaction 
for professional development services associated with the following programs: Smart Star, Smart Step, Next Step 
programs. Instruction Technology, Teacher Center Services, Instructional Materials and Delivery Services, 
Assistance in Meeting Goals/Academic Performance, and Effectiveness of Professional Needs Development. 

Table 1 Clients Level of Satisfaction with Professional Development Programs (n = 6,698) 

Professional Development Programs Very Satisfied Satisfied Not Satisfied (n/a) 

n% n % n% n % 

ELL 4,005 5945 69 L0 974 145 1,640 24.6 

Reading Recovery 4,688 70.0 59 0.9 641 9.6 1,310 19.6 
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Professional Development Programs 


Very Satisfied 


Satisfied 


Not Satisfied 




(n/a) 




n 


% 


n 


% 


n 


% 


n 


% 


ELF 2 


4,429 


66.1 


62 


0.9 


828 


12.4 


1,379 


20.5 


Next Step Strategies 


4,367 


65.2 


77 


1.1 


855 


12.8 


1,399 


20.8 


Literacy Labs 


4,137 


61.8 


79 


1.2 


909 


13.6 


1,573 


23.5 


Coaches Training 


4,549 


67.9 


77 


1.1 


642 


9.6 


1,430 


22.4 


Reading First 


4,482 


68.4 


84 


1.3 


681 


10.2 


1,351 


20.2 


Math-LINKS 


4,582 


68.5 


69 


1.0 


728 


10.9 


1,319 


19.7 


Geographic Information Systems (GIS) 3 


5,092 


76.0 


28 


0.4 


426 


6.4 


1,052 


17.2 


Math Coach Training 


4,795 


71.6 


56 


0.8 


525 


7.8 


1,322 


19.7 


Algebra, Geometry 


4,758 


71.0 


90 


1.3 


645 


9.6 


1,205 


18.0 


Math Content 


4,102 


61.2 


123 


1.8 


1,079 


16.1 


1,394 


20.8 


Smart Start, Smart Step, Next Step, etc. 


2,018 


30.1 


358 


5.3 


2,947 


44.0 


1,365 


20.5 


Instructional Technology 


964 


14.4 


713 


10.6 


3,315 


49.5 


1,706 


25.4 


Early Childhood Programs 


4,091 


61.1 


18 


2.1 


1.352 


20.2 


1.117 


16.6 


Career and Technical Education Services 


3,484 


52.0 


268 


4.0 


1.809 


27.0 


1.137 


17.0 


Gifted and Talented Services 


3,766 


56.2 


309 


4.6 


1,597 


23.8 


1,026 


15.4 


Health and Wellness Services 


3,535 


52.8 


303 


4.5 


1,829 


27.3 


1,031 


15.4 


Special Education Services 


3,391 


50.6 


350 


5.2 


1,797 


26.8 


1,160 


17.3 


ADE 4 Pro Dev Through Distance Learning 


3,542 


52.9 


358 


5.3 


1,760 


26.3 


1,038 


15.5 


Instructional Distance Learning for Students 


4,443 


66.3 


201 


3.0 


1.168 


17.4 


886 


13.2 


Teacher Center Services 


1,417 


21.2 


350 


5.2 


3,179 


47.5 


1,752 


26.1 


Instructional Materials & Delivery Services 


1,261 


18.8 


259 


3.9 


3,167 


47.3 


2,011 


30.1 


Services for Superintendents and Principals 


4,400 


65.7 


63 


0.9 


1.088 


16.2 


1.147 


17.1 


Assistance in Meeting Goals/ Academic Perf. 


1,363 


20.3 


415 


6.2 


3,359 


50.1 


1,561 


23.3 


Teacher Effectiveness and Learning 


926 


13.8 


461 


6.9 


3,682 


55.0 


1,629 


24.3 


Effectiveness of Professional Needs Develop. 


430 


6.4 


748 


11.2 


3,551 


53.0 


1,969 


29.4 


Assistance with Data & Support of ACSIP 5 


2,365 


35.3 


390 


5.8 


2,642 


39.4 


1,301 


19.4 


Support of APSCN 6 Services/Training 


3,700 


55.2 


268 


4.0 


1,587 


23.7 


1,143 


17.1 



Notes: ’Early Literacy Learning in Arkansas; Effective Literacy for Grades 2-4; 3 Geographic Information Systems; 4 Arkansas Department of 
Education; 5 Arkansas Comprehensive School Improvement Plan; 6 Arkansas Public School Computer Network. 



Reasons ESCs are Not Utilized 

Table 2 indicates reasons why clients may not have utilized the Arkansas ESC for their professional 
development. Thirty percent (n = 1,679) indicated the distance to the program sites was the reason for not utilizing 
the ESCs, while 24 percent (n = 1,314) indicated a lack of interest in accessing the services. Further, 16 percent 
(n=659) indicated a concern for the quality of service providers and 12 percent (n = 650) did not believe there was a 
need. The balance of the clients were unfamiliar with the programs (9%, n = 511), were not needing the services 
(12%, n = 650), or indicated they lacked the time (4%, n = 211) to utilize the services. 



Table 2. Reasons Why Clients May Not Have Utilized ESCs for Professional Development (N = 5,296) 



Reason Service or Program Was Not Utilized by Client 


Frequency 


Percent 


Lack of Time 


211 


4 


Lack of Interest 


1,314 


24 


Distance to Program Sites 


1,679 


30 


Type of Offerings 


272 


5 


Quality of Service Providers 


659 


16 


Lack of Need 


650 


12 


Unfamiliar with the Programs (new teacher to the area) 


511 


9 



Qualitative Results 



In an open ended item, respondents were asked for specific changes or improvements that would improve the ESC’s 
services. A total of 503 participants responded to this question. The most predominant area for improvement that 
emerged from analysis of the textual data was technology. This item was most frequently cited as needing 
improvement. The recommendations associated with technology improvement fell into three categories, quality, 
scheduling, and relevance. Figure 1 displays each of the three areas for improvement associated with technology and 
provide a summary of the respondents’ textual narrative that describes their rationale for suggesting change. 

Table 3. What changes/improvements would you recommend for ESC to further your professional goals ? 

Suggestions Rationale 

Technology - - The technology training that I have been to both at your facility and others have been obsolete. Better telecommunications 
Quality technology - Satellite programs are sketchy at best 
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Suggestions 



Rationale 



- Lots of technology problems with the DLL trainings 

- 1 would like more advanced technology for the classroom. I came for PPT training and was needing further training to take it to 
the next level, i.e., attaching music files, making sure when copying the PPT that the music files copy with it. I specifically 
asked the instructor for information on this and he never covered it for us. He did show us how to attach or insert the music, 
but never how to make it attach continually. 

- Improve the Technology Day. There are too many people for such a limited space for each topic 

- Offer more technology items, but make the classes smaller so that each can receive more 1 on 1 instruction 

- Co-op needs to be cutting edge with offerings of technology and assisting with technology training 

- Technology day is a waste of time. The sessions are too short and often crowded 

- A couple of times when I've come down for technology training, the computers . . . something was wrong with them 
Technology - More technology workshops offered during the summer... this is a huge problem. 

Scheduling - . . .. workshops I don't really need and can't use because the others are already filled up. 

- Very few technology courses are offered. If so, they are offered during the school year which causes a burden on the school to 

hire substitute teachers. Therefore, I rarely apply for programs during the school year 

- As the state mandates a set number of hours for technology, it would be most beneficial if our Co-op offered MORE technology 

P.D. courses throughout the summer. Those are the first ones that fill up because they are the ones that EVERYBODY must 
have 

- Offer more technology through the year, not just summer and more in the summer with larger classes 

- More advanced technology in-services during the summer 

- Not enough useful 1-day technology workshops, descriptions of workshops should be more accurate. Too many workshops 

scheduled during school year instead of summer break 
Technology - 1 personally need technology workshops in how to use the PowerPoint Program 
Relevance - More Elective Course technology training, We need more technology/math/science offerings 

- Gear more in-service toward high school and technology, Workshops for advanced technology 

- Real life classroom management, and useful technology workshops 

- Give us something to create and at the same time let us learn how it works. All of us know that we learn best with guidance. We 
want the same thing 

- Technology classes should be specifically geared toward different levels of expertise. It doesn't help me to be in classes with 
those whose technological skills are far superior to mine. I usually come away frustrated. One particular technology instructor 
has very little time to devote to those who are not already "on track". It would be far more beneficial to offer simpler classes for 
those who need them, than to place everyone into a "one size fits all" class. 

The second open ended question asked respondents to list four areas of professional development that the 
Cooperatives might offer to better serve the needs of the educational community. Figure 2 displays each of the four 
professional development areas and includes a summary of the respondent’s textual narrative that describes their 
rationale for these suggestions. The textual narrative is representative of typical responses to the question. 

Table 4. Areas of professional development that the Co-op might offer to better serve your needs 

Concern 

Areas Rationale 

Technology - How to implement use of technologies in the classroom. How to design class websites 
Education - How to use the Distance Learning Equipment at local schools 

- Network protection and Presentation of new technology and software 

- How to use the technology in the classrooms. We have good technology and the teachers know how to operate the equipment— 

-but they struggle with how to work that into the lessons they are currently doing. 

- In-service for the Media Specialists during the summer. Our school does not count the meeting during the school year toward our 

required in-service hours 

Math - More in-services in the technical aspects of science related to science frameworks. 

Science - Labs for the biology classroom that correspond to the frameworks, EOC Biology 
Education - Biology pacing guides , Science labs that are inexpensive 

- High school reform including in-service for high school teachers to become more student centered. 

- Science is going to need a lot of support. Elementary Teachers need help with easy labs in order to meet the 20% requirement 

- They need support in the "Why" something works 

- Lego Engineering workshops. We need more hands on physics and engineering 

- Quick Labs for Science, and more professional development in math for teachers 
Special - More Classroom management courses. 

Education - More on Autism and how to deal with these children, Behavior Modifying 

- More about Reading First for Special Education 

- Migrant student information/ ESL / subpopulations 

- Specialized training in helping students with learning disabilities, such as dyslexia 

- Response to Intervention- I'd like a roundtable type discussion on what other schools do for pre-referral interventions 

- For GEN ED TEACHERS - Speech/language therapy necessary? AND What a Language Delay/Deficit is? To answer the ever 

popular question, "Why is he going to speech, he can talk just fine!", 

- Dealing with behavior in the regular classroom (ADD, ADHD, OCD, ODD, Autism and all the others....) 

Literacy - Literacy Corners and other kindergarten literacy workshops, Reading instruction for non core teachers 
Education - Teaching high school teachers how to teach reading with emphasis on instructional reading. 

- Reading / Literacy, Arkansas Reading First , ESL/ELL Training 

- Any Literacy/Writing to help address benchmarks/EOC 
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Discussion and Conclusions 



The primary purpose of this research was to conduct a client satisfaction survey of more than 6,629 ECS 
stakeholders. The respondents' perceptions represent an audit of the level of performance of the Arkansas Education 
Services Cooperation as perceived by the clients. From the survey findings it is concluded that respondents were 
very satisfied with the professional development services in all of the programs offered within two of the three 
categories studied: (1) Literacy Programs (ELLA, Reading Recovery, ELF, Next Step Strategies, Literacy Labs, 
Coaches Training, Reading First,) and (2) K-12 Maths Program/Training (MathLINKS, GIS, Math Coach Training, 
Algebra, Geometry, Math Content). Further, in the third category. Other Professional Development Programs, it is 
concluded that respondents perceived moderate to high levels of satisfaction in eight professional development 
programs (Early Childhood Programs, Career and Technical Education Services, Gifted and Talented Services, 
Health and Wellness Services, Special Education Services, ADE Professional Development Services Through 
Distance Learning, Instruction Distance Learning For Students, Services for Superintendents and Principals, and 
Support of APSCN Services/Training), and low to very low levels of satisfaction for seven professional 
development programs: Smart Start, Smart Step, Next Step, etc,. Instructional Technology, Teacher Center Services, 
Instructional Materials and Delivery Services, Assistance in Meeting Goals/ Academic Performance, Teaching 
Effectiveness and Learning, and Effectiveness of Professional Needs Development. 

The researchers also concluded that most of the clients utilized the ESCs services 3 to 4 times a year, with less 
than half (40%) frequently using the services and less than 8 percent never using the services. Further, 
approximately 30 percent of the respondents do not use the ESCs due to the long distance, while approximately 24 
percent indicated a lack of interest. Contrary to ESC 2003 study, clients indicated they allocated more time to utilize 
the ESCs and became more familiar with the ESC programs. It was also concluded that the majority of the 
respondents were teachers, while other respondents were principals or assistant principals, superintendents, 
classified workers or clients affiliated with the main offices of their institutions. Further, it is concluded that 
approximately 44 percent of the respondents were employed for more than 10 years, 24 percent were employed for 5 
to 10 years, 17.26 percent were employed from 2 to 4 years, and 14.5 percent were employed less than 2 years. Also, 
approximately 61 percent of the respondents were employed as elementary, junior high, and middle school teachers 
and 25 percent were employed as high school teachers. The findings of this study are subject to certain limitations 
since the study primarily focused on a single element of summative evaluation, more commonly referred to as level 
one of Kirkpatrick’s (1998) four-level evaluation model. The primary criterion for the survey was clients' perception 
of satisfaction based on a Likert-type scale. This limits the amount of data and conclusions that may have otherwise 
been captured with a comprehensive systematic evaluation approach (Preskill & Russ-Eft, 2006; Scriven, 1995). The 
qualitative results of the study are also limited since there were no follow up interviews, focus group discussions and 
document analysis. 

Implications for HRD and Recommendations for Future Research 

The findings of this study show that HRD practitioners involved with evaluation work need to invest ample 
resources, primarily time, in conducting evaluation. Clients need to be educated that evaluation goes beyond 
satisfaction surveys. Russ-Eft and Preskill (2005) indicated that when critical factors are overlooked during the 
process of evaluating HRD initiatives, the data may be of little use or invalid. While the results of this study may be 
of some use to the ESC administrators, a more comprehensive evaluation is recommended. A systems framework 
model would “consider a number of variables that may affect not only the design of the evaluation but its 
implementation and the extent to which, and the ways in which, the evaluation findings might be used” (Russ-Eft & 
Preskill, 2005, p. 73). Thus, the emphasis should be to frame any evaluation so that the effects of all organizational 
and environmental factors are captured and accounted for when evaluating outcomes of an HRD initiative or 
program. Further, the results imply an acute awareness or focus of evaluation concerning several components is vital 
to the evaluator’s success. For example, evaluators/practitioners should fully understand the purpose of the 
evaluation, the relevant stakeholders, and essential questions that serve to guide and perhaps limit the study. Russ- 
Eft & Preskill (2005) indicate one means of attaining focus is to “convene a group of stakeholders who have a 
vested interest in the program being evaluated, are intended users of the results, or are potential future recipients of 
the program or service being evaluated...” (p. 75). With regard to Arkansas’ ESCs, there is no empirical evidence of 
record indicating a comprehensive evaluation has ever been carried out. Research emphasis should be placed on the 
utilization of a systems-based logic model of evaluation where external influences such as workforce diversity, legal 
ramifications, client expectations, innovations in technology, and competition and global influences are considered 
in alignment with the ESC’s vision, mission, and strategic plan. Additional future research should focus on ESC’s 
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infrastructure. This should include, but not be limited to the culture, leadership, systems and structures, and 

communication systems that potentially influence an evaluation design and utilization of the evaluation findings 

(Russ-Eft & Preskill, 2006). 
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